perm filename ESW.LET[AM,DBL] blob
sn#515270 filedate 1980-06-07 generic text, type T, neo UTF8
4-Jun-80 11:46:55-PDT,20077;000000000001
Mail-from: ARPANET site SUMEX-AIM rcvd at 4-Jun-80 1144-PDT
Mail from RAND-UNIX rcvd at 4-Jun-80 1142-PDT
From: Abh at Rand-Unix
Date: 4 Jun 1980 at 1142-PDT
To: lenat at Sumex-Aim
cc: Rick at Rand-Unix, Mary at Rand-Unix
Subject: Draft letter and outline
Doug, even though the "To" or "cc" doesn't mention it, Don did get
a copy of this message; it was sent with your message to SCORE.
Don and Doug -
Here is my draft letter and outline. Note especially weak sections
(e.g., 4.1, 5.4).
Doug: Please send revisions by Friday.
Don: Please send out letters Monday.
I will be out of town next week.
Rick
*****
Expert Systems Workshop Revised Outline and Instructions, 6/2/80
The attached outline presents an expanded statement of the six major
areas for the workshop groups and papers. You have several tasks:
1. Familiarize yourself with the overall outline.
2. Generate a position paper in response to this outline for your
group leader by July 1, 1980.
3. Contribute evidence or references to answer, support, or refute
the assertions and questions in the outline.
More detail on these activities follows below. We hope you enjoy this
preparation for the workshop. The better the early contribution, the
better we anticipate the workshop and resulting book papers will be.
Let us know if you have additional suggestions.
TASK 1: Familiarize yourself with the whole outline. We believe this
outline will develop into a lucid statement of the knowledge in the
field. Thus, we hope it seems cogent and integrated. It has many
weaknesses currently, but these should be eliminated by the time of
the workshop. The workshop goals are twofold: (1) we want to write
down the current wisdom so that hundreds of new people can make use of
it for the applications; (2) we want to formulate clear definitions of
frontier problems for the leaders in the field to address in their own
work. Everyone has a stake in the pursuit of these two goals.
TASK 2: Generate a position paper by July 1. Every invitee to the
workshop will belong to a group for codifying and communicating the
knowledge in one of these six areas. To insure that the workshop
achieves its potential, we expect the group leaders to prepare refined
section outlines before arriving in San Diego. The workshop will
begin with presentations on these refined outlines. Thus, by the
time you arrive at the workshop, considerable progress should have
already been made. Your initial responsibility is to assist in the
reformulation of the field's wisdom. Please address each of the
points in the draft outline in a way that elucidates your views of the
technology and science. Provide your leader specifics, examples,
precise assertions, references, taxonomies, statistics, concepts or
solutions! The more lucid your inputs, the better the revised
outlines and the more educational the workshop. Good luck and good
shooting!
TASK 3: Contribute evidence on references. The outline attempts to
survey the whole field of expert systems. As you scan other sections
of interest to you, you will undoubtedly find opportunities to
contribute nuggets of wisdom to other groups. Please send these along
to the appropriate group leader. We're sure the more inputs they
receive, the happier and more thorough they will be. Please send all
references possible via Arpanet in the bibliographic format of
Computing Surveys. The group leaders should collate these and prior
to the workshop, we will generate reference lists for all
participants. This should encourage the early development of a
comprehensive bibliography. We hope the excitement we feel in
anticipation of this workshop has begun to infest you too! We have
lined up a very interesting mystery expert, and we look forward to the
opportunity to integrate several knowledge engineering experiences
with the groups' scholarly presentations of ideas.
We look forward to seeing you.
Rick Hayes-Roth Don Waterman Doug Lenat
Attachment 1:
Addresses of the six group leaders
Ron Brachman RBRACHMAN@BBN-TENEXD
Bolt Beranek and Newman, Inc.
50 Moulton Street
Cambridge, MA 02138
Bruce Buchanan BUCHANAN@SUMEX-AIM
Stanford University
Department of Computer Science
Stanford, CA 94305
Mark Stefix STEFIX@SUMEX-AIM
Stanford University
Department of Computer Science
Stanford, CA 94305
David Barstow BARSTOW@SUMEX
Schlumberger
Old Quarry Road
Ridgefield, CT 06877
Randy Davis KRD@MIT-AI
Massachusetts Institute of
Technology
Artificial Intelligence Laboratory
545 Technology Square
Cambridge, MA 02139
Pete Hart HART@SRI-KL
SRI International
Computer Resources
333 Ravenswood Ave.
Menlo Park, CA 94025
Attacment 2: Expert Systems Workshop 6/2/80, Revised Outline
1. Definition of Expert Systems
1.1 Formal and informal characterizations of expert systems
1.1.1 Expertise
What is it? Domain model plus problem-solving schemes
organized efficiently
An expert system is an implemented body of expertise
1.1.2 Knowledge
Precise definition: concepts, facts, theories,
constraints, heuristics, ...
How is a knowledge-based system different from a Fortran
problem-solving procedure?
Is expertise=domain knowledge plus problem-solving
knowledge?
1.1.3 Tasks
Expert systems operate in narrow domains. Why?
How is the domain modeled in or reflected by the ES?
1.2 Alternative and contrasting methods
1.2.1 Analytical
These methods are systematic and rigorous
Examples include admissible searches or exhaustive searches
These methods should be verifiable, if slow
What kinds of algorithms for ES tasks have been developed?
What new algorithms seem ripe for development?
1.2.2 Heuristic
These methods are errorful, incomplete or uncertain
Examples are limited search, confidence-rating schemes,
or guesses
These methods should be fast, if risky
What kinds of systematic heuristic search techniques are
useful?
What class of heuristic methods aren't yet systematized?
1.2.3 Relative advantages of analytical vs. heuristic
Can you place error bounds on heuristic methods?
Can you compare heuristic to analytic solutions?
Is knowledge fundamentally heuristic?
How much computing is necessary to move from heuristic
to analytic solutions? Is it tractable/feasible?
1.3 The niche of expert systems
1.3.1 Expertise exists
Areas that exemplify/don't fit this requirement
What kind of expertise? Symbolic, expressible, testable,...
1.3.2 Knowledge and heuristics required
What kinds of problems don't require one or both of
these?
Why aren't such tasks appropriate for ES applications?
1.3.3 Payoff exceeds cost
What does it cost to build an ES?
What is its real life-cycle cost?
Can we measure the benefits?
(See criteria for performance 6.1)
1.3.4 What kinds of problems seem to fit into this niche?
Which don't? Why not?
2. Knowledge Acquisition
2.1 Extracting knowledge from experts
Typical experiences at this process
Usually involves programmer plus expert and produces hybrid
people
Why is it hard?
What techniques make it easy?
2.2 Conceptualization
2.2.1 Choices of key concepts and relations
2.2.2 Choices of representation schemes
2.2.3 Relationships between alternative representation schemes
2.2.4 Reasons for favoring a particular representation
2.2.5 Mapping the conceptualization into the representation
2.3 Assimilating new knowledge
What triggers the recognition for the need?
What are the schemes for adding increments of knowledge?
What kinds of increments cause difficulty?
When is the conceptualization thrown out?
How do we design a knowledge system to increase its flexibility,
visibility?
2.4 Extensibility of conceptualizations
What concepts/inference methods are hard-wired into the system?
What kinds of extensions arise during the course of a project?
What language/system/features enhance or impede extensibility?
What limitations arise? (E.g., combinations, inconsistencies?)
2.5 Consensus formation
What is the definition of a consensus?
How do we compare/contrast alternative formulations?
How do we average, aggregate or retain multiple formulations?
What methods exist for promoting or guiding consensus formation?
2.6 Automated discovery
What methods exist for generalizing the current knowledge?
Do any means exist for automatically identifying all
possible situations that may arise and eliciting the
appropriate response?
Can a repetoire of desired behaviors be inferred from examples
or by automated experimentation?
Has new knowledge ever been discovered in an expert system?
How can we limit the erroneous bits of knowledge automated
schemes introduce?
Which classes of knowledge seem well-suited to current automated
discovery methods? Which classes are on the frontier today?
3. Architecture
3.1 Logical or plausible foundations
Propositional or predicate calculus
Theorem proving or causal reasoning
Causal models
Model-based interpretation of observables
Plan generation with model-based evaluations
3.2 Inference and control
3.2.1 Deductions
Necessity
Sufficiency
Constraints
Causes/antecedents
Effects/consequents
3.2.2 Certainty calculations
Uncertainty as potential error
Reducing uncertainty by adding evidence
Schemas for combining uncertainties or evidence
Other forms of continuous judgments (e.g. Berliner's
BKG, Hearsay II priorities, AM "interestingness")
3.2.3 Search
Spaces of hypotheses/alternatives/partial solutions
3.2.4 Control
Combinatorics => search
Lack of combinatorics or lack of deep inferences => no search
Ways to simplify/reduce search
(e.g., abstraction, constraint propagation)
Serial schemes (e.g., AM, Hearsay II)
Parallel schemes (e.g., relaxation, back chaining)
Plateaus, optima, horizons and other phenomena
Time vs accuracy trade-off (e.g., how does
deeper search affect performance?)
3.3 Problem-solving structure
Multi-level (Hearsay II, Hearsay III)
Back chaining (MYCIN)
Homogeneous production rules (ROSIE)
Semantic nets (Internist)
Others(?)
3.4 Incomplete searches
3.4.1 Resource allocation
What principles do we have to chosing where to work?
When is an inference redundant?
What mechanisms do we use (agendas, meta-planning
or scheduling)
How much does this cost? What does it save?
3.4.2 Order of search
Which things should be computed first?
How does the order affect the outcome and its validity?
How can we measure the error of an incomplete search?
How can we know when to terminate?
Can we limit our errors somehow?
How can we assess the efficiency of the search (what
is prumed?)
4. Knowledge Engineering
4.1 Current tools and methods for acquisition
Knowledge representation packages (KRL, UNITS, ROSIE, KAS,...)
Interactive programs for knowledge base creating and testing
Top-down development styles (successive elaboration of
knowledge)
4.2 Current tools and networks for implementation
Programming systems for expert systems
EMYCIN, ROSIE, OPS4, Hearsay III, AGE, Expert
Top-down development style
Explanation/graphics for viewing system behavior
Test procedures
Debugging methods/packages
4.3 Matching tasks to tools
Characterize the tools and especially the differences among them
Use these characteristics to induce task environment distribution
Which task environment suggest which tools?
4.4 Weaknesses of current tools and methods
What limitation of the tools force us to restrict the tasks?
Knowledge acquisition is extremely ad hoc
Testing, evaluation, and knowledge refinement is also ad hoc
Few of the systems are both general programming langauges and
user-oriented
Most of the systems have particular niches owing to their
historical backgrounds
Most tasks permit radically different encodings with different
ramifications; i.e., we have insufficiently constrained the
design options for ESs
5. Meta-cognition
5.1 Self-description and monitoring
How can an expert system understand its own behavior?
How should it detect inconsistencies or errors?
What requirements are there for self-monitoring
Do any current systems support these capabilities?
Which additional capabilities are most important?
What would these take to implement? What would they cost?
5.2 Explanation
5.2.1 Schemes
Back chaining or forward chaining explanation by
trace interpretation
English vs code: what's important, and can we tolerate
separate explanatory text and code (i.e., just use
comments)?
Macro-scopic approaches to explanation, e.g., disease
prototypes or event standard scenarios (scripts)
Hence, multi-level approaches to explanation
Others?
5.2.2 Organization of explanations: which text comes first?
5.2.3 Formal underpinnings of explanation theory
5.2.4 When would an ES fail to be able to generate an
explanation? Or, are these systems complete AND
consistent?
5.2.5 Methods for explaining or portraying multiple simultaneous
plausible alternatives
5.3 Evaluation
5.3.1 What are the qualities requiring evaluation?
Accuracy
Certainty
Completeness
Effort required
Likelihood of eventual success
Priority of information to gather or tasks to execute
Impact of potential action
Credibility of a conclusion
5.3.2 Ad hoc methods for evaluation
Examples and experiences
5.3.3 Formal methods for evaluation
Advantages
Weaknesses and limitations
5.4 Reorganization
5.4.1 Knowledge reorganization
What triggers a need for reorganization?
What levels of reorganization have occurred? Why?
(e.g., Hearsay I -> Hearsay II; AM -> Eurisko;
DIALOG -> Internist)
What are the needs for self-awareness in these systems?
5.4.2 Problem-solving structure reorganization
Does this occur?
5.4.3 Reorganization of inference and control
Do different interactive discourses suggest different
control regimes?
When would a system need to restructure its control
strategy?
Do any systems currently do this? What's needed?
6. Performance
6.1 Criteria of success
6.1.1 Task performance (objective achievement)
6.1.2 Competence (behave as an expert)
6.1.3 Systematization/axiomatization of knowledge/methods
6.1.4 Codification of knowledge/methods
6.1.5 Extensiveness/completeness of search in uncertain domain
6.1.6 Robustness under uncertainty (comprehensiveness,
flexibility)
6.1.7 Credibility of decisions
6.1.8 Accountability of knowledge/methods
6.1.9 Maintainability/modifiability/extensibility
6.1.10 Naturalness of inference process
6.1.11 Speed and resource requirements (e.g., core, processor)
6.1.12 Human-machine interface
6.1.13 Verifiability/validity of knowledge/inference method/and
implementation
6.1.14 Reliability, continuity of decisions (insensitivity),
reproducibility
6.2 Measuring - Methods for measuring performance gainst each criterion
6.2.1 Problem-solving ability: Test and Evaluation
6.2.1.1 Familiar problems with known solutions
6.2.1.2 Problems with unknown but predicted solutions
6.2.1.3 Measuring individual units of knowledge
6.2.1.4 Measuring overall decisions
6.2.2 Ability to replicate expert decisions (on benchmark
problems or new ones)
6.2.3 Logical or formal analysis of capabilities
6.2.4 Representability and intelligibility of knowledge
6.2.5 Assessing the thoroughness or incompleteness of
heuristic searches
6.2.6 Does the system peruse alternatives and maintain their
entailments or does it make arbitrary choices? How
do these choices affect correctness of decisions?
6.2.7 Are the decisions correct? Are the reasons self-evident?
What are the sources of credibility for the decisions?
What arbitrary actions/choices are made that can introduce
error? Are these apparent or explainable to the user?
6.2.8 How can we assign responsibility for good/bad decisions to
elements of the knowledge base? How can we assess the
criticality or correctness of continuous variables?
6.2.9 How easy/hard is it to assimilate knowledge? What kinds of
additions cause fundamental restructuring?
6.2.10 Does the inference scheme match the expert's or is it
counter-intuitive?
Do explanations follow or mask the actual decision
procedures?
6.2.11 How fast does the system run? How is speed related to
amount of knowledge or depth of inference?
6.1.12 How easy/hard is it for a human to guide or understand the
system?
How well does the system exploit human channels (touch,
vision, natural language,...)?
6.1.13 Can you verify or validate the system? How? What kinds of
systems would be certifiable?
6.1.14 How can you measure the smooth, continuous response space of
the system over slowly changing assumptions? How critical
are various arbitrary parameters or combining rules?
6.3 Consistency
6.3.1 Is the implemented knowledge consistent with human beliefs?
Is it more specific in some ways? More abstract?
6.3.2 Are all inferences this knowledge generates consistent
with human beliefs, or do they sometimes produce strange
effects?
6.3.3 Do different experts in this field have consistent
beliefs? How are their differences reflected in the system?
6.4 Debugging
6.4.1 How do we recognize errors in a system?
6.4.2 How do we assign responsibility for errors?
6.4.3 What kinds of knowledge/inference units can be altered
and how? What kinds can't be? Why?
6.4.4. What methods for systematic testing exist?
6.4.5 What kinds of specification/verification are appropriate?
6.4.6 What special facilities for programmer debugging can
be provided?
6.5 Credit Assignment
6.5.1 How do we identify which knowledge elements are
Invalidated by data
Useful
Often valid
Often incorrect
Certain
Uncertain
Efficient
6.5.2 What changes to knowledge/inference/control should be made
to reflect each of these changes?
6.5.3 Can we identify macro-inferences or chains that operate
collectively, frequently, and usefully? How would we
change the systems to reflect his "chunk?"
6.5.4 Can we identify which sources of knowledge (e.g., different
experts) tend to provide the most valid/useful knowledge?
What can we do to exploit this recognition?
6.5.5 Can we adjust the program's asserted "certainties" to reflect
actual variability in decisions under equivalent
circumstances?
6.6 Interactive discourse style
6.6.1 What is the protocol or style of human-machine cooperative
problem solving with this system? How does this differ
and excel over the human-only method?
6.6.2 Natural language
6.6.3 Graphics
6.6.4 Explanation/question answering
6.6.5 Human-guided search
6.6.6 Interactive data collection strategies
6.6.7 Interactive styles of rule development and testing